Preparing lessons: Improve knowledge distillation with better supervision

Knowledge distillation (KD) is widely applied in the training of efficient neural network. A compact model, which trained to mimic representation a cumbersome model for same task, generally obtains better performance compared with being ground truth label. Previous KD-based works mainly focus on two aspects: (1) designing various feature knowledge transfer; (2) introducing different mechanism such as progressive learning or adversarial learning. In this paper, we revisit standard KD and observe that teacher’s logits might suffer from incorrect uncertain supervision. To tackle these problems, propose novel approaches deal respectively, are called Logits Adjustment (LA) Dynamic Temperature Distillation (DTD). be specific, LA rectifies according label certain rules. While DTD treats temperature dynamic sample wise parameter rather than static global hyper-parameter, actually notes uncertainty each sample’s logits. With iteratively updating temperature, student could pay more attention samples confuse teacher model. Experiments CIFAR-10/100, CINIC-10 Tiny ImageNet verify proposed methods yield encouraging improvement KD. Furthermore, considering simple implementations, can easily attached many frameworks bring improvements without extra cost time computing resources.